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ABSTRACT 

A study conducted in the intensive English language 
program at Indiana University investigated: (1) the effects of 
students 1 language learning background (formal learning of English, 
exposure to and use of English in class, and extracurricular exposure 
to and use of English) on their performance on the Test of English as 
a Foreign Language (TOEFL); and (2) students 1 perceptions of their 
current language learning needs and of language instruction in their 
home countries. Subjects were 101 students from a variety of native 
language backgrounds. Results suggest that two background factors 
(extensive outside reading and high teacher proficiency in English) 
have a significant positive effect on TOEFL performance, and that 
communicative oral use of English in and outside of class affects 
TOEFL performance through its positive influence on outside reading 
behavior. Student observations also highlighted the importance of 
proficient and qualified teachers. It is concluded that these 
criteria should be incorporated in evaluation of English programs, 
focusing less on specific test results than on the conditions that 
promote the ability to function in the target language. (MSE) 
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PROGRAM EVALUATION IN LIGHT OF LANGUAGE 
LEARNING BACKGROUND , STUDENT ASSESSMENTS 
AND TOEFL PERFORMANCE 
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INTRODUCTION 

Wc have carried out a study of the language learning background of students in our 
intensive English program to identify variables which have a positive effect on students* 
TOEFL performance. Data were collected from over 100 students in the program and the 
information coded and statistically analyzed. Using multiple regression and path analysis, we 
developed a model showing direct and indirect effects of a number of background variables 
on students* TOEFL scores. These variables, which include communicative use of English in 
class and extensive outside reading, suggest aspects of language teaching which may be used 
in program evaluation. The study also investigated the students* perceptions of their current 
language learning needs and their suggestions for improving language teaching in their home 
country. The students* assessments and the results of the statistical analysis provide useful 
guidelines for the evaluation of ESL programs at home and abroad. 

At the Center for English Language Training at Indiana University, we have been 
interested in identifying factors in our students* language learning background which have 
a significant effect on language proficiency. Our main focus is students from other 
countries who have come to the United States to pursue higher education at American 
Universities. For the purpose of admission to academic university programs, the language 
proficiency test that is commonly used in the United States is the TOEFL (Test of English 
as a Foreign Language). We have therefore investigated background factors in relation to 
performance on the TOEFL examination, using as a reference point the initial TOEFL 
scores which the students obtained upon entering our Intensive English Program. We also 
looked into our students* perceptions of their language learning needs and their 
suggestions for improving language teaching in their home countries. 

In this paper, wc bring to bear findings from our research on the evaluation of ESL 
programs at home and abroad. We will first outline our procedure and summarize our 
findings, and then wc will consider the implications of these findings for program evaluation. 
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PROCEDURE 

The data for the study were collected by individual interview from iOI students in our 
Intensive English Program at Indiana University. The students came from a variety of first- 
language backgrounds, about equally distributed between Arabic, Japanese, Romance, and 
other languages. They had learned English in a formal environment to varving degrees, and 
their initial TOEFL scores with us were normally distributed. The background language 
learning information was elicited through an oral questionnaire and fell into four major 
categories: formal learning of English, exposure to and use of English in class, exposure to 
and use of English out of class, and attitudes and motivation. The items of information were 
coded and quantified, resulting in 44 background variables whose effects on the TOEFL 
scores were to be examined. Students' observations on aspects of their language learning 
background were also elicited and categorized. 

The students' scores on TOEFL were used as a measure of language proficiency. 
TOEFL is a standardized instrument for language assessment which is widely used by 
universities in the United States for admission purposes. The examination is comprehensive 
\J and covers a range of language abilities, from elementary to advanced levels. Foreign 

J students commonly must attain a specified minimum score as a prerequisite for admission to 
university work. The students in our Intensive English Program I Ac an institutional version 
\ i of the TOEFL at the end of each seven-week ses\ion. This icn, msists of three sections: 

**- Listening Comprehension, Structure and Written Expression, and Vocabulary and Reading 

Comprehension. 
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Three types of statistical analysis were used to examine the effects of background 
factors on language proficiency. The first was pairwise correlations between all the variables 
and the TOEFL scores, which led to the selection of a set of promising variables for closer 
examination. The second was multiple regression analysis, which further identified variables 
with a significant effect 

on TOEFL. The third was path analysis on a basic structural model, which showed both 
direct and indirect effects. 



RESULTS OF THE QUANTITATIVE ANALYSIS 

The first type of analysis was pairwise correlations between all the variables. Table 1 
shows the background variables which correlated significantly (p=.05) with TOEFL 
scores. 



Table 1 Background variables which correlated significatly 
with the TOEFL scores (p= .05) 



Extracurricular (outside) reading 


I 

53 


Native speaking teachers 


39 


English as language of instruction 


36 


Months of intensive/special English 


.26 


Effective teachers 


.21 


Future need for English 


21 



For a list of the variables, see Appendix 1 



As can be seen from Table 1, the highest correlation coefficient was obtained for 
extracurricular reading. It may also be noted that the first four variables seem to reflect the 
extent of exposure to and use of English for meaningful communication. The last two are 
related to attitudinal and motivational factors. Surprisingly missing from this list are 
classroom variables, such as index of oral exposure, index of oral use, and index of writing 
use, as well as the extracurricular variables of writing, speaking and listening, all of which 
involve the use of English in communication. However, examination of the full 
intercorrelation matrices revealed that the above variables that did not correlate directly with 
TOEFL, nonetheless correlated significantly with outside reading, which correlated well with 
TOEFL. This suggested the presence of indirect effects, which were later explored through 
path analysis. 

Based on the patterns of intercorrelation, 22 background variables were selected for 
further examination through multiple regression analysis. The results of this analysis, using 
the stepwise forward procedure (selection criterion p= .05), are given in Table 2. 



Table 2 Multiple regression analysis for 22 background 
variables with the TOEFL scores 



Steps 


R 


R2 


1 Extracurricular reading 


.53 


.28 


2 + Native speaking teachers 


.59 


.35 


3 + Total index of exposure 


.64 


.40 (neg. beta) 


4 + Extracurricular speaking 


.66 


.43 (ncg. beta) 


All 22 variables (backward procedure) 


.75 


.57 



For a list of the variables sec Appendix 1 




As can be seen in Table 2, the single most important background factor affecting 
performance on TOEFL is outside reading, followed by exposure to teachers who are native 
speakers of English. The two factors combined account for over one third of the variance in 
the TOEFL scores (R2=.35). The 22 variables together account for well over half this 
variance (R2=.57). The seemingly negative effects of total index of exposure and 
extracurricular speaking will in the next analysis (path analysis) be seen to conceal positive, 
though indirect, effects. 

Path analysis was undertaken in order to explore the indirect relationships observed in 
the pairwise correlations and the unexpected negative effects observed in the multiple 
regression results. In this type of analysis, a causal model is formulated, consisting of a set of 
variables with interconnecting paths indicating the direction of effects. The statistical 
procedures calculate coefficients for the effects and evaluate the model by determining the 
extent to which it fits the available data. The statistical program we used for this purpose is 
LISREL (Analysis of Linear Structural Relations). 

To date, we have constructed a basic model comprised of six variables: two classroom 
variables, three extracurricular (or outside class) variables, and the TOEFL scores. The 
paths lead from the classroom variables to the outside of class variables, and from these to 
outside reading and to TOEFL. The model showing the variables and the selected paths, all 
unidirectional, is illustrated in Figure 1. Preliminary results from the application of the 
statistical program LISREL to the full model indicated that a number of paths in the model 
did not represent statistically significant effects. The model was therefore trimmed to 
eliminate these relationships from the equation. The resulting modified model, showing the 
remaining paths and their coefficients, is presented in Figure 2. 
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Figure 2 

Modified Structural Equation Model Shoving Significant Paths 
and their Coefficients 

Goodness of fit index - .98 



As can be seen, reading outside class again shows a strong direct effect on the 
TOEFL scores. The only other variable in this model that has a direct effect on TOEFL is 
listening outside class, but this relationship is a negative one. Speaking outside class, which 
had a negative relation with TOEFL in the multiple regression analysis, now shows a positive 
effect on TOEFL through its effect on reading. Likewise, the two classroom variables, oral 
exposure to English and communicative oral use of English, are now seen to affect TOEFL 
indirectly. Both have paths with positive direct effects on outside speaking, which in turn 
leads through outside reading to TOEFL. Oral exposure, in addition, has a path to outside 
listening, which has a negative relationship with TOEFL. 

What this model seems to indicate is that the single most important factor in 
improving proficiency as reflected in the TOEFL scores is outside reading. Extent of oral 
exposure and communicative use of English in class and out of class have a positive, though 
indirect, effect on TOEFL in so far as they promote outside reading, but not through their 
effect on outside listening. Outside listening, which reflects the extent to which learners were 
exposed to English speech through radio, television, or film, appears to have a negative 
relationship with TOEFL. 
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Two aspects of these findings require further comment: the negative relationship 
between outside listening and TOEFL, and the strong positive relationship between outside 
reading and TOEFL. 

Concerning outside listening, it should be noted that in our study the scores for this 
variable do not necessarily reflect active listening. In most cases, the students reported that, 
while viewing English programs, they relied totally on the native language subtitles and did 
not pay attention to or were finable to understand E ng lis h speech. This type of exposure, 
therefore, did not represent comprehensible input, unlike the active listening included in the 
classroom variable, oral exposure. However, this explanation still leaves unaccounted for the 
negative relationship of outside listening with TOEFL scores. One possible interpretation is 
that students with higher outside listening scores may have a tendency towards passive 
viewing at the expense of more demanding activities, such as reading and social 
communication. 

As for outside reading, the results of the study clearly appear to indicate that reading 
for personal information or pleasure is a more important means of implicit learning than 
exposure to and use of spoken English in and out of the classroom. It may be argued that 
the prominence of this factor reflects the degree to which good performance on the TOEFL 
depends on reading ability. While this may partly be the case, it seems unlikely that the use 
of reading as a medium for this examination can adequately account for the strength of the 
relationship of outside reading with TOEFL and its subsections. It would seem more likely 
that extensive outside reading helps to improve the level of proficiency in a global sense, 
enhancing acquisition of grammar, vocabulary, and rhetorical structure, as well as increasing 
the geneval knowledge base which helps in reading comprehension.! 

In order to explore this point further, we ran multiple regression analysis using only 
those students in our sample who had entered at the three upper levels of our program 
(levels 4, 5, and 6) on the assumption that, for this more advanced group, basic reading skills 
had already been acquired. The results of this analysis (not tabulated) confirmed our finding 
for the whole sample. Outside reading was once again the most important, indeed the only, 
factor with a significant effect on TOEFL scores. Those students who read extensively out of 
class within this more advanced group attained higher levels of proficiency. 

Another type of analysis that we applied to the data gave further support to the 
importance of outside reading in enhancing language proficiency. Using the t-test, we 
examined differences in the mean TOEFL scores of students who had done some outside 
reading and those who had not. We found that, even among students who had had the 
benefit of studying in an intensive English program, the mean scores of those who had done 
outside reading were significantly higher than those who had not. 

The dominant role of outside reading which emerged from this study, although 
unexpected, is perhaps not surprising. Elley and Mangubhai (1983) in their experimental 
study on the effect of an extensive reading program on the language development of students 
in a number of Fijian primary schools found that students exposed to extensive reading of 
high-interest story books made significatly greater gains in language skills than the control 
group. In a more recent study, Tudor and Hafiz (1989) found that extensive reading 
improved students' writing significantly, particularly the level of accuracy. These findings are 
in accord with the theoretical viewpoint put forward by Krashen that reading, by providing 
extensive comprehensible input, is an important and effective means of acquiring language 
(Krashen 1981, 1988, 1989). 

To sum up, the background factors which were found to have a direct positive effect 
on TOEFL scores were outside reading and teachers who were native speakers of English. 
Exposure to and use of spoken English in class and out of class for communicative purposes 
seem to be helpful only in so far as they promote outside reading. Although our present 
path analysis model does not include the variable nat ; "e speaking teachers, it is reasonable to 
expect that a strong relationship hoids between this tactor and extent of exposure to and use 
of spoken English for communication in class and out of class. Indeed, pairwise correlation 
coefficients between these variables and native speaking teachers were quite high, ranging 
between .42 and .46. 



RESULTS OF THE STUDENTS' OBSERVATIONS 

Another aspect of the language learning background rtiat we examined in this study 
was based on the students' qualitative observations. Durhg the interview, we elicited from 
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the students their comments on four aspects of their language learning experience: (1) why 
they liked or did not like their English class in high school; (2) what characteristics they 
valued in teachers of English whom they remembered as particularly good; (3) what they 
perceived to be their present language learning needs; (4) what their suggestions would be 
for improving the teaching of English in their home countries. Their responses, which were 
revealing, are outlined in this section. 

Concerning the students' attitude towards learning English, about 45% of the students 
said that they had* liked their English class when they were in school The reasons they gave 
were fairly evenly distributed among the following categories: 

(1) They thought English was useful or important. (27%) 

(2) They liked English as a language, liked its sounds, liked to speak it. (20%) 

(3) They regarded English class as an enjoyable period in which they could 
participate actively. (20%) 

(4) They liked the teacher. (16%) 

(5) They did well in English. (16%) 

The reasons students gave for not liking English class represent the other side of the 
coin and are at least equally interesting. The reasons given, in order of frequency (high to 
low), are as follows: 



(1) English was too difficult; they could not understand it and did poorly in it. (40%) 

(2) The English class was boring, not interesting, a waste of time. (32%) 

(3) They did not like the teacher. Reasons given included that the teacher was too 
mean, too strict, was not interested in teaching, could not speak or pronounce 
English well. (19%) 

(4) English was not useful to them; it was imposed on them. (17%) 

(5) The material taught was not relevant to their needs or interests. For example, 
they wanted to learn to speak but they were taught only grammar and reading. 
(6%) 

These observations are echoed in the characteristics of teachers of English whom the 
students remembered as being particularly good. About 80^ of the students said that they 
had had memorably good teachers. The characterizations that these students gave are listed 
below in the order of their frequency of mention: 

(1) The teacher explained well, had a clear purpose, and was serious about helping 
the students learn English. (39%) 

(2) The teacher had pleasing personality traits. Descriptions included: friendly, 
kind, encouraging, interesting, and having a sense of humor. (32<^>) 
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(3) The teacher used a variety of interesting activities and materials and encouraged 
the students to use English. (27%) 

(4) The teacher spoke English well, had good pronunciation, and used English in 
class. (23%) 

(5) The teacher made the students realize the importance of English in their lives 
and for their futures. (3%) 
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The next question concerned the students' perceptions of their current language 
needs. Of course, one must keep in mind that the students' responses reflect the fact that 
they are living in an English-speaking environment now and that most of them are planning 
to continue their education in American universities. It is therefore not surprising that the 
vast majority of the students felt that they needed more practice in listening and speaking. 
However, in addition, about 40% of the students mentioned the need to improve their 
reading and 6% their vocabulary through reading; about 28% recognized a need to improve 
their writing; about 9% felt they needed grammar, and about 4% said they needed to study 
hard. Two observations may be noted in connection with these results. The first is that our 
students' needs included all the language skills, singly or in various combinations, which 
confirms to us that none of these skills is superfluous in our program. The second is that 
there were important individual differences between the students, based on their past 
backgrounds and 

their future purposes, and that an intensive English program would do well to incorporate a 
measure of flexibility that would accommodate these differing individual needs. 

In the last question, we asked our students to suggest how the teaching of English 
could be improved in their home country. Of course we realize — and many of the students 
pointed this out - that the teaching of English has changed dramatically over the past few 
years and that many improvements have been introduced. Nonetheless, the suggestions 
made by the students, based on their own experience and perceived needs, do provide 
valuable criteria for evaluating English language programs. 

Most students called for the extensive use of English in English class and for 
increased attention to listening and speaking. Many stressed the importance of having 
teachers who are highly proficient in English, either native speakers or teachers who have 
received their professional education in English-speaking countries. Another set of 
suggestions relate to the interest factor in the classroom: varying the teaching methods and 
materials to include songs, games, tapes, movies, and reading analysis rather than mere 
translation. It was also suggested that reading and discussion should be on topics of current 
interest, and that the grammar should be of practical value rather than consisting of abstruse 
rules that are memorized with the object of passing exams. Several students also expressed 
the opinion that teachers should help their students recognize the importance of English in 
the modern world, encourage their efforts to learn the language, and make allowance for 
differences in individual ability within the class. We should perhaps add here that we were 
struck by the consistency in the observations the students made about their language learning 
experiences and their preferences, regardless of their native language and educational 
background. 

To conclude, in our study we used the TOEFL examination scores of our students to 
help us identify aspects of their language learning backgrounds which contribute to language 
proficiency. We found that two background factors have a significant effect on TOEFL: 
extensive outside reading, and teachers who have an excellent command of English. We also 
found that communicative oral use of English in class and out of class affects performance 
on TOEFL through its positive effect on outside reading. These findings, along with the 
students' observations, which tend to highlight the importance of proficient and qualified 
teachers, provide a set of useful criteria for evaluating English teaching programs based, not 
on how students do on a particular test, but on the conditions that seem to promote the 
ability to function in the target language. 



NOTES 

1 For the linguistic knowledge and complex processes involved in reading 
comprehension, see Goodman 1988, Carrell 1988, and Eskey 1988. For the relationship 
between general language competence and reading proficiency, see Ellcy 1984, 
and Devine 1988. 
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Appendix 1 

Background Variables Considered 



Formal Lfr arn^g fif English 

♦Age at start of English 

Years of English in school 

Years of English in University 

♦Months in intensive or special English programs 

Months of private English tutoring 

Contact hours for each of the above 

♦Total contact hours 

Years since last English class 

♦Type of schooling: private/public 



Exposure to and Use o f English in Class 

♦Teachers who are native speakers of English 
♦English as language of instruction 
♦French or Spanish as language of instruction 
Index of exposure to oral English 
Exposure to instructions 
Exposure to explanations 
Index of oral use 

Oral use - sentence practice 
♦Oral use communicatively 
♦Index of writing 

Writing - sentence practice 
Writing communicatively (composition) 
Index of communicative use (oral + written) 
Audio-visual index 
Listening to tapes 
♦Language Lab 
♦Total index of exposure 

(index of exposure + listening to tapes) 
Teaching focus 

(grammar translation/grammar reading/all skills) 
Type of intensive/special English progr am 
(home country/ES teachers/ES country) 



Extracurricular Exposure to and U sft of English 

♦Listening 
♦Reading 
♦Speaking 
♦Writing 

Travel to English speaking countries 
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Attitude an d Motivation 



•Attitude to English class in high school 

•Recognition of the need for English when in high school 

♦Effective teachers of English 

•Family encouragement for learning English 

•Family graduates from English speaking countries 

•Current English use out of class 

Current reason for intensive English 

•Future need for English on return to the home country 

* = the 22 variables selected for multiple regression analysis 



